Zabbix Alerts not working but I can send a manual test call

#1

Using Zabbix 5.0 docker build with ports 80 and 443 open. I can verify inside the container I am able to
curl https://app.pagerduty.com/webhook_ips
[“34.210.57.30”,“34.210.242.134”,“52.34.208.156”,“34.202.21.89”,“34.239.229.93”,“34.231.45.166”,“44.242.69.192”,“52.89.71.166”,“5
4.213.187.133”,“35.86.21.47”,“52.88.94.18”,“44.238.89.29”,“54.241.68.46”,“54.176.72.216”,“54.177.81.67”,“13.56.49.27”]

Problem: Setup a PagerDuty zabbix user account that should send all alerts under media for the Pager Duty webhook and unable to get this working.

When trying a manual run for the PagerDuty Webhook I am able to get alerts with entering a eventid, event_source, and eventvalue.

Name PagerDuty
Type Webhook
Parameters
Name Value Action
alert_message {ALERT.MESSAGE}
eventack {EVENT.ACK.STATUS}
eventdate {EVENT.DATE}
eventid {EVENT.ID}
eventname {ALERT.SUBJECT}
eventtags {EVENT.TAGS}
eventtime {EVENT.TIME}
eventupdate {EVENT.UPDATE.STATUS}
eventvalue {EVENT.VALUE}
event_source {EVENT.SOURCE}
hostip {HOST.IP}
hostname {HOST.NAME}
severity {EVENT.NSEVERITY}
token cc--------------------------aeb
triggerdesc {TRIGGER.DESCRIPTION}
triggerid {TRIGGER.ID}
triggeropdata {EVENT.OPDATA}
url {$ZABBIX.URL}

I have even tried the agent and was unsucessful.

#2

Are you following this? https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/media/pagerduty?at=release/5.4

Did you import the example yaml file?

You will be sending to the Events API so check those IPs to allow. More info here: https://support.pagerduty.com/docs/safelist-ips

#3

Thank for the info. We are using zabbix 5.0. When I try to import the yaml file I get;

  • Unsupported import file extension “yaml”.

I did import a xml file for a zabbix 5.0 instructions. media_pagerduty.xml

#4

I seen this > v1 Webhooks EoL (End of Life) is next month

This is a follow-up reminder to previous communications. The End of Life date for v1 Webhooks is 10/31/2022. This means:

  • You won’t be able to create new v1 Webhooks or use existing connections to v1 Webook extensions.
  • Apps or integrations that are using v1 webhooks will stop working.

For more details and steps to migrate to v3 Webhooks please refer to this page. If you have additional questions, please reach out to support@pagerduty.com.

#5

Zabbix “webhook” is not the same as a PagerDuty Webhook (v1/v2/v3), so that is not related. You’re creating PD Events using our Event API v2 format.

Can you reach https://events.pagerduty.com/v2/enqueue from your Zabbix 5.x server? Can you run a simple cURL command from there and successfully create a PD Event outside of Zabbix?

#6

bash-5.0# curl -kv https://events.pagerduty.com/v2/enqueue

  • Trying 44.233.86.211:443…
  • Connected to events.pagerduty.com (44.233.86.211) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  • ALPN, server accepted to use http/1.1
  • Server certificate:
  • subject: C=US; ST=California; L=San Francisco; O=PagerDuty, Inc.; CN=*.pagerduty.com
  • start date: Mar 3 00:00:00 2022 GMT
  • expire date: Apr 3 23:59:59 2023 GMT
  • issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
  • SSL certificate verify ok.

GET /v2/enqueue HTTP/1.1
Host: events.pagerduty.com
User-Agent: curl/7.79.1
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 405 Method Not Allowed
    < Server: nginx
    < Date: Tue, 04 Oct 2022 18:22:55 GMT
    < Content-Length: 18
    < Connection: keep-alive
    < cache-control: max-age=0, private, must-revalidate
    < x-request-id: FxrwzQPNTLh3bRcISG_D
    <
  • Connection #0 to host events.pagerduty.com left intact
    Method Not Allowed
#7

I read this about setting a trigger

I am not sure that this relates to what I am doing because the alert should be sent from the user I created in Zabbix. I am not sure about setting an action for a trigger. These seem to be for email notifications.

#8

OK i tried using Events API V2 and Zabbix Webhook (for 5.0 and 5.2). I am able to get the test to work. I also tried to setup an action inside zabbix.

ACTION
Name PagerDuty Notifications
Type of calculation: And/Or
Conditions:
Trigger severity equals *Disaster
Trigger severity equals High
Trigger severity equals Average
Trigger severity equals Warning
Trigger severity equals Information

Operations
Send message to users: dmoore (David Moore), PagerDuty User via PagerDuty
Recovery operations
Send message to users: dmoore (David Moore), PagerDuty User via PagerDuty

Still is not working.

Looking in Zabbix logs I do not see any action from pager failing except when I did a manual test without filling in paramaters. Seeing this showed me how to send a manual test from media portion of zabbix;
root@ps-prod-docker-zabbix:/usr/local/docker# docker logs -f 436f2ebbca63 | grep PagerDuty
224:20220926:200845.239 [PagerDuty Webhook] Notification failed : incorrect value for variable “eventid”. The value must be a positive number.
223:20221004:140631.537 [PagerDuty Webhook] Notification failed : incorrect value for variable “eventid”. The value must be a positive number.

#9

I might have gotten this to work. Thank you to the people who posted the info for setting up an alert.

#10

I did get pager duty working with zabbix by setting up a alert with a trigger.